Papers by Éric de la Clergerie
MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization algorithms have been an essential component of language modeling but their static nature results in important flaws that degrade the models’ downstream performance and robustness. |
| Approach: | They propose a module for Adaptive Neural TokenizAtion that is differentiable and trained end-to-end with the language model. |
| Outcome: | The proposed tokenizer improves robustness to character perturbations and out-of-domain data. |
CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data (2024.lrec-main)
Copied to clipboard
| Challenge: | Clinical data in hospitals are unstructured and therefore need to be extracted from medical reports to conduct clinical studies. |
| Approach: | They propose a dedicated French biomedical model based on a public French biomedicine dataset. |
| Outcome: | The proposed model improves 2.54 points of F1-score on biomedical named entity recognition tasks. |
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases (2022.lrec-1)
Copied to clipboard
| Challenge: | MUSS trains strong models using sentence-level paraphrase data instead of labeled simplification data. |
| Approach: | They propose a multilingual unsupervised sentence simplification system that does not require labeled simplification data. |
| Outcome: | The proposed model outperforms the previous best supervised models on English, French, and Spanish benchmarks despite not using labeled simplification data. |
CamemBERT: a Tasty French Language Model (2020.acl-main)
Copied to clipboard
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, Benoît Sagot
| Challenge: | Pretrained language models are now ubiquitous in Natural Language Processing, but their use in other languages is limited. |
| Approach: | They propose to train monolingual Transformer-based model for other languages using web crawled data instead of Wikipedia data and a relatively small web crawl dataset leads to better results. |
| Outcome: | The proposed model performs as well as those obtained using larger datasets. |
On the Scaling Laws of Geographical Representation in Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Language models embed geographical information in their hidden representations, but larger models cannot mitigate this bias. |
| Approach: | They propose to extend this finding to Large Language Models by observing how geographical knowledge evolves when scaling language models. |
| Outcome: | The proposed model scales consistently with increasing model size, but smaller models cannot mitigate geographic bias inherent in training data. |
ANCOR-AS: Enriching the ANCOR Corpus with Syntactic Annotations (L18-1)
Copied to clipboard
| Challenge: | ANCOR-AS is an enriched version of the ANCor corpus that adds syntactic annotations in addition to the existing coreference and speech transcription ones. |
| Approach: | They propose to use syntactic annotations in addition to existing coreference and speech transcription annotations to improve detection of mentions. |
| Outcome: | The proposed version adds syntactic annotations to existing coreference and speech transcription annotations and is released in a new TEI-compliant XML format. |
Controllable Sentence Simplification (2020.lrec-1)
Copied to clipboard
| Challenge: | Text simplification is often considered an all-purpose generic task where the same simplifications are suitable for all but multiple audiences can benefit from simplified text in different ways. |
| Approach: | They propose a controllable simplification model that provides explicit control on simplification systems based on Sequence-to-Sequence models. |
| Outcome: | The proposed model outperforms standard models on simplification benchmarks. |